Image sensor device with macropixel processing and related devices and methods

ABSTRACT

An image sensor device may include an array of image sensing pixels with adjacent image sensing pixels being arranged in macropixel, and a processor coupled to the array of image sensing pixels. The processor may be configured to receive pixel signals from the array of image sensing pixels, and arrange the received pixel signals into macropixel signal sets for respective macropixels. The processor may be configured to perform, in parallel, an image enhancement operation on the received pixel signals for each macropixel signal set to generate enhanced macropixel signals, and transmit the enhanced macropixel signals.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/730,312, filed Oct. 11, 2017, which application is a continuation ofU.S. application Ser. No. 14/836,143, filed on Aug. 26, 2015 (now U.S.Pat. No. 9,819,913), which applications are hereby incorporated hereinby reference.

TECHNICAL FIELD

The present disclosure relates to the field of integrated circuits, and,more particularly, to an image sensor device and related methods.

BACKGROUND

Typically, electronic devices include one or more camera modules forproviding enhanced media functionality. For example, the typicalelectronic device may utilize the camera modules for photo capturing andvideo teleconferencing. In the typical electronic device with multiplecamera modules, the primary camera module has a high pixel density andan adjustable focus lens system, while the secondary camera module isfront-facing and has a lower pixel density. Also, the secondary cameramodule may have a fixed focus lens system.

For example, U.S. Patent Application No. 2009/0057544 to Brodie et al.,assigned to the present application's assignee, discloses a cameramodule for a mobile device. The camera module comprises a lens, ahousing carrying the lens, and a lens cap over the lens and housing. Thecamera module includes a barrel mechanism for adjusting the lens. Eachcamera module comprises an integrated circuit (IC) image sensor devicehaving a plurality of pixels arranged in an array of rows and columns, aplurality of pixel lines coupled to the plurality of pixels, and readoutcircuitry coupled to the plurality of pixel lines.

Referring to FIG. 1, a typical electronic device 100 is now described.The electronic device 100 includes an image sensor array 101, aprocessor 102 coupled to the image sensor array, and a host device 103coupled to the processor. The processor 102 includes a receiver module104, a plurality of processing modules 105 a-105 n coupled in successiondownstream from the receiver module, and a transmitter module 106coupled downstream from the plurality of processing modules.

Referring to FIG. 2, another typical high performance (i.e. havinghigher pixel processing power) electronic device 200 is now described.The electronic device 200 includes an image sensor array 201, aprocessor 202 coupled to the image sensor array, and a host device 203coupled to the processor. The processor 202 includes a receiver module204, and a splitter module 207 coupled downstream from the receivermodule and for dividing the data into one or more stripes of data. Theprocessor 202 includes a plurality of pipelines coupled to the splittermodule 207, each pipeline comprising a plurality of processing modules205 a-205 n coupled in succession. The electronic device 200 includes amerger module 208 coupled downstream from the plurality of pipelines,and a transmitter module 206 coupled downstream from the merger module.

SUMMARY

Generally speaking, an image sensor device may include an array of imagesensing pixels with adjacent image sensing pixels being arranged inmacropixels, and a processor coupled to the array of image sensingpixels. The processor is configured to receive pixel signals from thearray of image sensing pixels, arrange the received pixel signals into aplurality of macropixel signal sets for respective macropixels, perform,in parallel, at least one image enhancement operation on the receivedpixel signals for each macropixel signal set to generate enhancedmacropixel signals, and transmit the enhanced macropixel signals.Advantageously, the image sensor device may process data moreefficiently and with a lower clock speed.

More specifically, the plurality of macropixel signal sets may comprisea plurality of Bayer macropixel signal sets. The image sensor device mayfurther comprise a Bayer color filter array over the array of imagesensing pixels. Each of the received pixel signals for each macropixelsignal set may represent an individual color. The processor may beconfigured to perform the at least one image enhancement operation oneach individual color.

For example, the at least one image enhancement operation may comprise abrightness control operation. Each macropixel may comprise a 2×2 set ofindividual pixels. The processor may be configured to merge the enhancedmacropixel signals into an image. The array of image sensing pixels maycomprise a complementary metal-oxide semiconductor (CMOS) image sensorarray.

Another aspect is directed to an electronic device comprising aprocessing unit and associated memory, and an image sensor devicecoupled to the processing unit. The image sensor device may comprise anarray of image sensing pixels with adjacent image sensing pixels beingarranged in macropixels, and a processor coupled to the array of imagesensing pixels and configured to receive pixel signals from the array ofimage sensing pixels, and arrange the received pixel signals into aplurality of macropixel signal sets for respective macropixels. Theprocessor may be configured to perform, in parallel, at least one imageenhancement operation on the received pixel signals for each macropixelsignal set to generate enhanced macropixel signals, and transmit theenhanced macropixel signals.

Another aspect is directed to a method of operating an image sensordevice comprising an array of image sensing pixels with adjacent imagesensing pixels being arranged in macropixels, and a processor coupled tothe array of image sensing pixels. The method may include operating theprocessor to receive pixel signals from the array of image sensingpixels, and operating the processor to arrange the received pixelsignals into a plurality of macropixel signal sets for respectivemacropixels. The method may include operating the processor to perform,in parallel, at least one image enhancement operation on the receivedpixel signals for each macropixel signal set to generate enhancedmacropixel signals, and operating the processor to transmit the enhancedmacropixel signals.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present invention, and theadvantages thereof, reference is now made to the following descriptionstaken in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic diagram of an electronic device, according to theprior art.

FIG. 2 is a schematic diagram of another electronic device, according tothe prior art.

FIG. 3 is a schematic diagram of an electronic device, according to thepresent disclosure.

FIG. 4 is a schematic diagram of an array of image sensing pixels with aBayer filter from the electronic device of FIG. 3.

FIG. 5 is a detailed schematic diagram of an electronic device,according to the present disclosure.

FIG. 6 is a schematic diagram of an image enhancement operation modulefrom the electronic device of FIG. 5, according to the presentdisclosure.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present disclosure will now be described more fully hereinafter withreference to the accompanying drawings, in which several embodiments ofthe invention are shown. This present disclosure may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein. Rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the present disclosure to those skilled in theart. Like numbers refer to like elements throughout.

Referring initially to FIGS. 3-4, an electronic device 10 according tothe present disclosure is now described. The electronic device 10illustratively includes a processing unit (e.g. multi-core system onchip device) 12 and associated memory 13, and an image sensor device 14coupled to the processing unit. As will be appreciated by the skilledperson, the processing unit 12 may comprise a host device, such as amobile cellular device or a tablet computing device. The electronicdevice 10 illustratively includes a housing 11 carrying the image sensordevice 14, the processing unit 12, and the memory 13.

The image sensor device 14 illustratively includes an array 15 of imagesensing pixels 16 a-16 ff with adjacent image sensing pixels beingarranged in macropixels 18, and a processor (i.e. the image signalprocessor (ISP)) 17 coupled to the array of image sensing pixels. Forexample, the array 15 may comprise a complementary metal-oxidesemiconductor (CMOS) image sensor array. The processor 17 and the array15 of image sensing pixels 16 a-16 ff may comprise a single IC, or inthe alternative, each component may be on separate ICs.

As perhaps best seen in FIG. 4, each macropixel 18 illustrativelyincludes a 2×2 macropixel. Of course, in other embodiments, themacropixel 18 may have other dimensions, such as 3×3, 4×4, 2×4, etc. Theimage sensor device 14 illustratively includes a Bayer color filter overthe array 15 of image sensing pixels 16 a-16 ff. Of Course, in otherembodiments, the image sensor device 14 may comprise other types offilters alternatively or additionally. In particular, the plurality ofmacropixel signal sets comprises a plurality of Bayer macropixel signalsets. Each of the received pixel signals for each macropixel signal setillustratively represents an individual color. In Bayer filterembodiments, each macropixel set comprises a red signal R, a blue signalB, and first and second green signals G1, G2.

Referring now additionally to FIG. 5, the processor 17 illustrativelyincludes a receiver module 28 configured to receive pixel signals (i.e.data) from the array 15 of image sensing pixels 16 a-16 ff. The pixelsignals may be formatted as a raster scan pattern image, but otherformats can be used. The processor 17 illustratively includes amacropixel converter module 19 configured to convert the pixels signalsfrom the receiver module 28 into a plurality of macropixel signal sets,each macropixel signal set being for a respective macropixel 18. Theprocessor 17 illustratively includes a plurality of image enhancementoperation modules 20 a-20 n (defining an image enhancement pipeline),each configured to perform a respective image enhancement operation ofthe plurality of macropixel signal sets.

As illustrated, the processor 17 is configured to perform, in parallel,the plurality of image enhancement operations on the received pixelsignals for each macropixel signal set to generate enhanced macropixelsignals. In other words, in the illustrated embodiment, the four Bayersignals (R, B, G1, G2) of each macropixel signal set are processed inparallel. Additionally, the processor 17 sequentially feeds theplurality of macropixel signal sets into the plurality of imageenhancement operation modules 20 a-20 n. Hence, each image enhancementoperation module 20 a-20 n of the image enhancement pipeline is workingon a single macropixel signal set at a time in each clock cycle period.

The processor 17 illustratively includes a macropixel converter module21 downstream from the plurality of image enhancement operation modules20 a-20 n and configured to covert the sequential plurality ofmacropixel signal sets into formatted image data (i.e. back to theoriginal format before division into macropixels 18). The processor 17illustratively includes a transmitter module 22 downstream from themacropixel converter module 21 and configured to transmit the enhancedmacropixel signals as formatted image data (e.g. raster image data).

Referring now additionally to FIG. 6, an example image enhancementoperation module 20 is shown. Here, the image enhancement operationmodule 20 comprises an image gain operation circuit (i.e. brightnesscontrol). The image enhancement operation module 20 is configured toperform the image enhancement operation on each individual color for theplurality of macropixel signal sets, i.e. an input 29. The imageenhancement operation module 20 illustratively includes configurationregisters 23 for storing operational parameters, and a gain circuit 24coupled to the configuration parameters. The gain circuit 24illustratively includes a plurality of mixers 26 a-26 d respectivelyreceiving the signals for each macropixel signal set, and controller 25controlling a data path 30. The gain circuit 24 is configured to outputan enhanced plurality of macropixel signal sets, i.e. an output 31.Advantageously, the gain circuit 24 may apply individual gain factors toeach color signal.

Another aspect is directed to a method of operating an image sensordevice 14 comprising an array 15 of image sensing pixels 16 a-16 ff withadjacent image sensing pixels being arranged in macropixels 18, and aprocessor 17 coupled to the array of image sensing pixels. The methodincludes operating the processor 17 to receive pixel signals from thearray 15 of image sensing pixels 16 a-16 ff, and operating the processorto arrange the received pixel signals into a plurality of macropixelsignal sets for respective macropixels 18. The method includes operatingthe processor 17 to perform, in parallel, at least one image enhancementoperation on the received pixel signals for each macropixel signal setto generate enhanced macropixel signals, and operating the processor totransmit the enhanced macropixel signals.

In typical approaches, most CMOS image sensors capture images using theBayer color filter. This filtering may be processed before the image canbe displayed or encoded for storage or transmission. An ISP is used toprocess these images. With increasing resolution and/or increasing theframe rate, these images must be processed rapidly. This may require theISP to run at a very high clock frequency, which poses issues inphysical implementation and increases the power consumptionsignificantly. The herein disclosed electronic device 10 provides anapproach to a low power image processing pipeline, which processes thepixels captured by the sensors using Bayer color filters.

The input image for the ISP may be in the Bayer format, requiring apattern of 2×2 pixels to reconstruct the three color components. The ISPimplements various image enhancement operations by processing theindividual pixels. With increasing megapixels and/or the frame rate,there is a need to run these ISPs at a very high clock frequency. Forexample, moving from a 5 megapixel to a 16 megapixel imaging deviceoperating at 30 fps, the minimum clock frequency requirement changesfrom 150 MHz to 480 Mhz. Apart from posing the challenge for timingclosure, it also affects the power consumption. The later one is animportant issue that must be addressed for low power systems (i.e. thesystems that are “always on”, e.g. gesture recognition, security camerasand the systems that operate on battery, such as mobile phones).

In the typical approach of FIG. 2, the input image is stored in thememory and split in vertical stripes for processing. These stripes areprocessed independently and stitched together later to form a completeimage. This concept can be extended for power saving and instead of asingle processing pipeline running at a very high frequency, multiplepipelines can be used to reduce the operating frequency. For example,the illustrated four pipelines will allow the maximum clock frequency tobe reduced to 25% of the original frequency. This may allow the designto be implemented at a lower supply voltage, and the design can beclosed with higher VT cells. Due to this, both the dynamic range andcurrent leakage may be reduced, and low power implementation is madepossible.

Advantageously, the electronic device 10 also reduces the operationalclock speed to 25%, but with reduced circuit resources. Also, in thedisclosed electronic device 10, the power consumption depends on theswitching rate. By processing a macropixel 18, similar color pixels areprocessed through the pipeline as opposed to all colors using samepipeline in other approaches. In natural images, the change betweenadjacent pixels of same color is less than that of the adjacent pixelsof different color. Due to this, the dynamic power consumption will beless in a macropixel processing block than having four parallelpipelines working on Bayer image, as in the approach of FIG. 2.

Moreover, the duplication of logic circuitry is reduced in theelectronic device 10, as compared to the parallel architecture as theconfiguration registers 23, for example, are not replicated. This mayreduce the silicon real estate consumed by the processor 17. In theelectronic device 10, programming overhead may be small because thereare no stripes to manage (as in FIG. 2). In particular, in the approachof FIG. 2, the offsets must be correctly programmed for each stripe.Also, the electronic device 10 is more intuitive and fits naturally withsome processing blocks, which work on macro-pixels to reduce theartifacts.

Many modifications and other embodiments of the present disclosure willcome to the mind of one skilled in the art having the benefit of theteachings presented in the foregoing descriptions and the associateddrawings. Therefore, it is understood that the present disclosure is notto be limited to the specific embodiments disclosed, and thatmodifications and embodiments are intended to be included within thescope of the appended claims.

While this invention has been described with reference to illustrativeembodiments, this description is not intended to be construed in alimiting sense. Various modifications and combinations of theillustrative embodiments, as well as other embodiments of the invention,will be apparent to persons skilled in the art upon reference to thedescription. It is therefore intended that the appended claims encompassany such modifications or embodiments.

What is claimed is:
 1. An image sensor device, comprising: an array ofimage sensing pixels comprising a plurality of macropixels, wherein eachmacropixel comprises contiguous image sensing pixels; and a processorcoupled to the array of image sensing pixels, the processor beingconfigured to: receive data from the array of image sensing pixels;convert the data into a plurality of macropixel signal sets, wherein arespective macropixel signal set corresponds to a respective macropixelof the array of image sensing pixels, each macropixel signal setcomprising a plurality of individual signals; serially feed theplurality of macropixel signal sets as a sequence of macropixel signalsets into a plurality of sequential image enhancement operations; andperform, for a given macropixel signal set of the sequence of macropixelsignal sets, the plurality of sequential image enhancement operations onthe plurality of individual signals of the given macropixel signal setto produce enhanced macropixel signals, wherein a given operation of theplurality of sequential image enhancement operations operates inparallel on the plurality of individual signals of the given macropixelsignal set.
 2. The image sensor device of claim 1, wherein the processoris further configured to merge the enhanced macropixel signals into animage and to transmit the image.
 3. The image sensor device of claim 1,wherein the data is formatted as a raster scan pattern image of thearray of image sensing pixels.
 4. The image sensor device of claim 1,wherein the plurality of macropixel signal sets comprises a Bayermacropixel signal set.
 5. The image sensor device of claim 1, furthercomprising a Bayer color filter array over the array of image sensingpixels.
 6. The image sensor device of claim 1, wherein the plurality ofsequential image enhancement operations comprises a brightness controloperation.
 7. The image sensor device of claim 6, further comprising again circuit configured to execute the brightness control operation,wherein the gain circuit comprises a plurality of mixers, wherein arespective mixer is configured to apply a respective gain factor to arespective signal of the plurality of individual signals.
 8. The imagesensor device of claim 1, wherein the plurality of individual signalscomprises a plurality of individual color signals.
 9. The image sensordevice of claim 8, wherein the plurality of individual color signalscomprises a red signal, a blue signal, a first green signal, and asecond green signal.
 10. An electronic device comprising: a processingunit; a memory coupled to the processing unit; and an image sensordevice coupled to the processing unit, the image sensor devicecomprising: an array of image sensing pixels comprising a plurality ofmacropixels, wherein each macropixel comprises contiguous image sensingpixels; and a processor coupled to the array of image sensing pixels,the processor being configured to: receive data from the array of imagesensing pixels; convert the data into a plurality of macropixel signalsets, wherein a respective macropixel signal set corresponds to arespective macropixel of the array of image sensing pixels, eachmacropixel signal set comprising a plurality of individual signals;serially feed the plurality of macropixel signal sets as a sequence ofmacropixel signal sets into a plurality of sequential image enhancementoperations; and perform, for a given macropixel signal set of thesequence of macropixel signal sets, the plurality of sequential imageenhancement operations on the plurality of individual signals of thegiven macropixel signal set to produce enhanced macropixel signals,wherein a given operation of the plurality of sequential imageenhancement operations operates in parallel on the plurality ofindividual signals of the given macropixel signal set.
 11. Theelectronic device of claim 10, wherein the plurality of individualsignals comprises a plurality of individual colors.
 12. The electronicdevice of claim 10, further comprising a Bayer color filter array overthe array of image sensing pixels.
 13. The electronic device of claim10, wherein each macropixel comprises a contiguous array of adjacentimage sensing pixels.
 14. The electronic device of claim 10, whereineach macropixel signal set comprises a red signal, a blue signal, afirst green signal, and a second green signal.
 15. The electronic deviceof claim 10, wherein the plurality of sequential image enhancementoperations comprises a brightness control operation, wherein respectivegain factors are applied to respective signals of the plurality ofindividual signals.
 16. A method, comprising: receiving data from anarray of image sensing pixels, the array of image sensing pixelscomprising a plurality of macropixels, wherein each macropixel comprisescontiguous image sensing pixels; converting the data into a plurality ofmacropixel signal sets, wherein a respective macropixel signal setcorresponds to a respective macropixel of the array of image sensingpixels, each macropixel signal set comprising a plurality of individualsignals; serially feeding the plurality of macropixel signal sets as asequence of macropixel signal sets into a plurality of sequential imageenhancement operations; and performing, for a given macropixel signalset of the sequence of macropixel signal sets, the plurality ofsequential image enhancement operations on the plurality of individualsignals of the given macropixel signal set to produce enhancedmacropixel signals, wherein a given operation of the plurality ofsequential image enhancement operations operates in parallel on theplurality of individual signals of the given macropixel signal set. 17.The method of claim 16, further comprising merging the enhancedmacropixel signals into an image, and transmitting the image.
 18. Themethod of claim 16, wherein the plurality of macropixel signal setscomprises a plurality of Bayer macropixel signal sets.
 19. The method ofclaim 16, wherein the plurality of sequential image enhancementoperations comprises a brightness control operation, wherein respectivegain factors are applied to respective signals of the plurality ofindividual signals.
 20. The method of claim 16, wherein each macropixelsignal set comprises a red signal, a blue signal, a first green signal,and a second green signal.